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abs!1!bact . '* ■ ' ' ' 

This paper discusses the problem of Selecting 
appropriate iDiniauii criteria for the review of educational programs ' 
and products that 'will not unduly hinder the creative freedom of 
educational developers. The author first asserts that although most 
people support the concept- of exanining and assessing educational 
products and programs, there is much' disagreement over pertineilt ' 
criteria. Although it jjfcossible to develop a matrix of crit;^ia for' 
the evaluation of each ^Rduct, the author suggests, the queltion of 
which criteria should be applied to all prxsdqcts should be/based on 
the free enterprise model.- That .is, criteria ^ould be ^plied" that 
assure the har^leSsness of a product, but beyond that point the 
aarJ^et place should be allowed *to work' its will free of further 
requirements. Specifically, the only .criteria that should be anplied 
to the ass,§«i»ent of any new educational product before its rexease, 
the a^lhor argues, are the criteria of "social balance and fairness'' 
an4 "user effects." (Author/JG) • . . . ~ 
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me begin* "the •individual presentations by discussing the criteria < 

x:eptable to the"^ducationai developer — the producer 'of these things we 

are so concerned about. * 

* * t 

When I think of the educational developer' s* role in this matter, I get 
a mental image of the developer, like Atlas of ancient mythology, trying to 
balance the bulk of ,a huge inverted pyramid of legislation, criteria, arid 
exf)ectations, while simultaneously striving to be imaginatives daring, ef~ 
ficient, and effective in creating a solution to an educational problem. 
If you develop a similar image as you listen to the presentations which fol- 
low, then, perhaps, you will understand why I expect to be the defensive 
one on th^ panel today. ' - * , 

Since ten minut.es is not very much time, let me just assert a couple 

of things at the outset. ^ - - ^ ^ 

■ ' ' ■ * N 

1. I endorse completely the concept of examining §nd assessing-^l^^ 

educational products and programs on a wide variety of bases, 
' but I i^elieve that until we have greater understanding of the 
consequences of employing these criteria, any effort to require 
ary large number of them would a huge errors. I'm afraid that 
such a move^ would signal the end of the very kind of careful, 
thorough development most of us are trying to encourage, becausel. 
it would make that" type of development 'much too expensive and too 
' time-consuming. ^ ^ , . . 

2. I support the efforts currently being made to 'compile* l/sts of 
criteria appropriate for" use in ,re^^ievring educational products 
an<i programs. If we are ever going to deal witii the creature, 
we must first know^ its dimensions and chara'cteristics. But I 
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. do, want to stress 'that we' are still a long way from consensus on 

what "^sCns-titutes a comprehensive and accurate list of the 
^ pertinent criteria, let alone consensus on which of them are of 
greater importance than others. ^ It is still very early in this 
particular ball game and anything can happen* 

To illustrate the point, I h^^ made available copies' 3?ife a 
list beii^g generated by Al^n Collir of Research for Better School 
Let me point out two things. First, there are differences ^T-* ' 
in^ the way Alan Coller has classified certain ^j^x^^eria and the 
way Sue Klein* has .cj.assif ied apparently thei^ same *tjpiteria. This 
is npt to suggest that one is right and the other ^^xing, but to 
illustrate that we aren't in agreement on how to classify review 
criteria^ let alone how to interpret^ rank order, or* apply them. 

Second, you will note that the Coller list is incomplete 
in two respects. Alan has taken jUst the first category 
(Adoptability) and carried it to greater levels of detail. ^That 
task still ^remains to be done on the remaining five categories 
(Intrinsic Quality,. Desirability, etc.). Note for yourselves * * 
the large number of criteria he has identified already (68 -by my 
count) and he hasn't yet finished the list! This illustrates t 
for me, at least, that t:he practical prdblems of applying 
criteria in the review' of educational products and* programs 
are too many, and *too great, to be imposed to any great extent 
on the educational developer now.* ^ ' ^ * 

The other way in which the Coller list is incomplete is -^hat 
each of the individual listings still must be Jback^d up with 
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definitions which are mutually exclusive but, in toto, all- 
encompassing, •with standards for judging each, and, if they are^ 
/ to be of practical benefit, with evaluation questions appropriate 

' • to be asked under each of them,, I think this, too, indicates 

i 

^ " how very far away* we are from that point where we can have some 

f 

' confidence in making recommendati-sns with regard to the criteria 
appropriate for use in reviewing educational materials and pro- - 
grams dnd how risky it would be to settle^in on a particular 
set of reqi(irement^s now. . / \ 

With that as my attempt tci illustrate the' magnitude of the problems 
involved in the revieV of, educational products and programs, let me move 
on and devote the remain4er of nq^ time to tl^e matter of which criteria I 
believe would be at least temporarily acceptable to the^ educational developer. 
As I do so, I will use Sue Klein's classification, rather than Alan CqileY^'s^x 
so. we will'haVe a cojnmon base for cUscussiont \ • * I 

•Deciding which of the many possible c^teri^ to include and exclii4e 
from a minimally acceptable set is a difficult task, as I'm su^e my 
associates on th^ panel will agree. That's because the decVsions are largely 

i 

idiosyncratic to each product and program. Whether or not "social acceptance" 

is an important criterion depends on whether you '.re exanvAning a classroom ^ 

curri'culuin product or a self-help training manual for administrators. 

^On a per~product basis, I would use a criterion jnatrix "to help identify 
^/ . 'if',' 
the criteria most appropriate for use. I would list in tfie^left column 

-the f ull^ rang-e of criteria which had been identified^ As the column heads 

across the top of the matrix I would list (1) target population, (2) type of 
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product, (3) sponscr^;* expecVations, (H) puppise (^f the_deveJ3ppmenta{ effort, 
(5) legislative mandate-s, etc. Appropriate rankings could be made in such 
a matrix for e^ch product being considered for review. 

In looking ^;t the -Question of which criteria ought'to be applied as a minimum 

V 

to all products, I follow the free enterprise model. That is, I believe 

developers can reasonably be requ/Lped to meet criteria which' assure the 

^ harmlessness of the product on potential users, but that from that point on 

.there should be no 'further requirements and the market place should simply 

be allowed to; work its will. ^Stated another way, I believe that government, 

distributors, and developers themselves have a responsibility to guard the 

publife- from harm, and should actively seek to meef that respdnsibiUty ,by 

establishing prudential criteria and standards, but that the dangers of 

proceeding beyond that point are greater than the protections which would 

be obtained, ^n my view, thpse dangers wou^'d take the form of urtdue govern- 

meryt influence, censorship, , a reduction iA available alteijnatives, and 

possibly t)ne cessation of large segments of the development industry under 

the burden^ of a moutain of regulation and legislation. 

** * _ . * 

If you will look with me at the, column heads on Sjie Klein'^s question- 

\ . ^ 

naa,re, I vilL indicate what I thik are the implications of application of 
the fjee enterprise model to the various criteria which appear there 
' Th^ first column, labeled "Desirability," contains criteria .most 

V /: ' : . 

appropria^te tc^ the pre- development, or- needs ass^6<fSlTie7rt, phase of a 
developmental effort.- ' .Such' matters as the 'need ^r.the product, demand 



for tjrTproduct, and so forth are appropriate for discussion at that 
point b'etween the potential sponsor and po-teA^lal developer, but, in my 

. . . V . \ ' ' 

•ppmioiH; are ,not pertinent to the question of Whether a completed program 



or pir'oducStlcsn 'or ought to- be released for 'general use. .1 would not -include 

any of these criteria among the minimum set t<5 be required for release of 

% ^ ' . ' /' . 

a )>r^duct or program. . - ^, ^ * ^ ^ 

The criteria in the secon4 column, under the heading^".f>racticability, " 

* ' * \ --^^ 

seem to me to^ apply to the mSrketability of tlie finis hed pro-am or product. 

Thefee are import^t considerations, all right, but whether oy not there is " 

anjassureA markat for^a program or product is, I submit, not crucial to 

^ whether they ought to be made available to the marketplace, and I would 

not include these ^riteria/among the minimum set.«.<>' ^ • 

Moving for a ir^jent to the extoeme right hand column ("Spinoffs"), 

I regard the effects li>te^here (model for other' work, contributes to , 



knowledge) as boijus effefcts ^ich might be obtainable, from p^iragrams or 
products specifically desigft^d to bl" "models.H or to test "a s'et of develop- 
mental hypotheses, but not pertinent to all programs or products an'd, 
iTherefore, not among the mimmum set of criteria which ought* to be applied 
tor a px;;ogram or product to -i^e released, ' ^ ^ . 

Some, but not all of the criteria 'ix( columns three, four, and five 

» • 

do seem to me to be" relevant to the matter under examijiatibn. In. column 
three, Suejias -included "tebhnical quality, of physical .features ofi mater iai^" 
and "attractivfeness (of materi-als)" under the heading "Intrinsic Quality." 
I would have- listed these in t}ne second column," an'd, as I have already 
indicated, I consider th^se. 'marketability criteria not applicable to the . 
release or not release decision. i . ■ ' 

The other criteria in column tjiree include one I would include among 
the minimum set: the criterion of social balance or fairness. More will 
be said about this in a few minutes. * * \^ 

/ ■ ■ ■ ^ ■ 
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Tbe r^naining- criteria in colmn 'three are ^'content'^accuracy and currency'* 
and. "instructional quarlity, coherence. of design, clarify, Cand) clear purpose." 



These seem to me^ to be.-matt«^^s pf prudential concern to the* sponsor of- the 
* ' ♦ * , . • 

product, and ones which most would hope would' be of cc^cem to curricula 
selection committees.* I submit they ougTit not be legislated ^as being among 
any required minifnurn set, however, because of (1) the lack 9f gen-el^al agree-' 
ment on'cont^t (e,g,, the differing theories on the creation, the Great 

* — 

^ Books^ versus "relevalfice" schools of thought), f^)'our inability to sub- 
stantiate empirically the virtues of various instructional designs, and 
* (3) the wide variations in j^u^g^e which exist throughout ' the country. 
. . I would exclude thejcx>i'teria in t^e forth coliimn, labeled "Product 
jDevelppment," because they have to do with\echnical aspects of the process 
a'ljd' not^the outcome. Some may believe th^t particular developmental 

^ • > ' 7 r • . * . 

methods or certain staff credentials have a strong bearing on the quality 
of the program o.i^ product whibh f3 produced, gut there is no evidence ^ 
^to ^upport such a belief, so procedua?al requirements ought not l?e -included 
among any minimum set of require* ed c;rit^ria,^^ ' . • • 

Finally, in the column labeled "User Ef f ects,"<, is the major crite^oiT^ 

, • • / / - - - ^ - 

I would endorse as appropriate to apply to any new program or product before 
its release. -'i- Evidence on the effects of the new development—primary .and 
seconda^^, anticipated and unanticip^t^ed, cognitive and effective, physical • 
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'*«"L'earner ver if icaticin,"' which is included in column four 5^ is a process for 
" product "improvement after its release, so I have excluded it from' this , ^ 
discussion of release criteria. . - " - . • \/ 

• * 
--The other criterion in columQ five, credibility of evidenSTe, is part- and ^ 
parcel^'of the evidence, in my view, and therefore not distinguishable as' . 
a separate criterion, Would anyone give^ credit ^or- a categorical re-, 
quirement hSkying been* met If the evidence offered were incr^dibie'^ 
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and DSVcholoKical — is a fair minimum requirement^ To pick up on an eaUier 
comment, I would 'i]hclude evidence on~ spcl^l balance, or fairness as another 
^one of the acceptable minimum requirements. I 'see this as actually being- - 

one of. the types of "user effects," because its impact is primarily as a 
■ -psychological effect, but I concede that the topic now ^las gained enough 
political ^overtones that it is not unreasonable to consider it'(social 
balance) as\sQ^terion a^a-rt from its> effects on users/ 

These are th.e 'minimum criteria, and the onl^ criteria, then, which 
I see as being legitimately required -of developers at the presen't t: 
Other criteria may be\required by sponsors or implici-t in the develop^ 
effort itself, on a product i?y product basis,' but to go beyond the application 
of user- effects criteria on a universal ba§is^ is, in my opinion, risky, W 
sensible, land a danger to the continued existence of a large segment- of 
the development industry as we noW know it. 
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Si^toi'cnt: Categories of Criteria Which ^iay be l.>t\J jr- 
Cha^e Process A:tivities or*Rinctions (frjpluding Pt^oUucts) 



1.0.0.^.0. • MoptaWlity.. 



A^-q)tability. 
Balances 
Convntibility. 
Lfpality. 
!x^gitimacy, 
pertinence. 



l.Z'.O.O.O. Exportability. 
1.2.1.0.0. Integrity; 

1.2.1.1.0. Coi^leteness. 

1.2.1.2.0. Ihiity. 
1.2.2.0^.0. Independency. 
1.2.3.0.0. Convcyability. > 
1.2.4.0^0. Availability <Df Inventio*/ 
1.2. S. 0.0. Availability of Servic«^ 
1.2.6.0.0. Traihability. 



1.3.0.0.0. Feasibilijty. 
1.3.1.0.0. Reasonableness of*^sts. 
.3.1.1.0. Economical Soundness. 
.3.1.2.0.' Cost Competiveness. 
.2^0.0. Affordability. 
.3.0.0. Implewentability. ^ 
.3.3.1.0.* Resource Availability. 
.3.3.2.0. Resource Reconknendations. 



1.3.3.3.J)^'^esource Requirements. 



1.4.0.0.t). Flexibility. 
1.4.1.0.0. Adaptability. 
1.4.^0.0. Diversity. 

1.5.0.0.0. Manageability. 
' 1.5.1.0.0. Operability 

l.S. 2.0.0. Integra tiVcness. 

1.5.3.0.0. Maintainability. 
1.5.3.1.0. Durability. 

1.5.3.1.1. Continuance. 

1.5.3.1.2. Sturdiness. 

• 1.5.3.1.3. Perpetuality. 
1.5.3.2.0. Utility. 
1.5.3.3.0. Correctiveness. 



3.0.0.0.0. Desirability. ; 

3.1.0.0.0. SijjUtficance. 
3.1.1.0.0. Problcja-Sin4>lo invact. 
3.1.2.0.0. Intenhft^. 

3.2.0.0.0. Rclcviince. ' , 
3.2.1.0/.0; Ccntrality. 
3.2.2.J0.0. Scope. 

3.. ^.0.0.0. Tinieliness. 

•* 

3.-4.0.0.0. Marketability. 
* 3.4.1.0.0. Adiance. ' " 
3.4.2. 0'.O. Coinpetitiveness. 

3.^.0.0.0. Sijrplfe In^^act. 

• • • 

3.6.0,0.0. Compatibility, ■ 

4.0.0.0.0. Effectiveness. 

4.1.0.0.0. Nature of Effects.- 
4.1.1.0.0. S{m of Effects. 
4.1.2.0.0. Quality of Effects, , 

4.2.0.0.0. Potency of r.ffect(s). • * 
. 4.2.^.0.0. Outputs/Effect i venes s . 
4.2.1.1.0. Magnitude of Effect(s). 
4.2.1.2.0. Magnitude of Anticipated Effects. 
4.2.1.3.0. Magnitude of Desired Effects. 
4.2.2.0.0. JXiration of Effect(s).. 
4. 2. 3.0.0.' Mcaningfulness of Effect(s). 

4.3.0.0.0^ Cost/Effectiveness. 
4.3.-1.0.0. MagnitudCyOf Effects. 
4.3.1.0.0. Real Costs. * , 

4.4.0.0.0. Interactive Effects. 
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2.0.0.0.0. Intrinsic Quality. ^ 

2.1.0.0.0. Appeal. 

2.2.0.0.0. ^ntentuAl AccUVacy^^ 
2.2.1.0.0.^ Currency. * 
2 J 2 . 2 . 0 . 0 . Correcthes s^. 
2.2.3JifO». Sufficiency. \ 

2.3.0.0.0. -Harrolessness. 

2.4.0.0.0. Instructional Quality 

2.5.0.0.0. Social Fajfness 

^ 2.6.0.0.0. Technical Quality. 

2.7.0.0.0, Uniqueness. 

2.8.0'. 0.0. Product Development Process Adequacy 

developed by Alan R. Coller i " 
Evaluation Specialist , 
Researdf for Better Schools, Inc.* 
February, 1976 
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5.0.0.0.0. Impact. 

5.1.0.p.O. Range of Effects. 
5.1.1.0.0. latent of Effects, 
5.1.2.0.0. Spread of Effects. 

5.2.0.0.0. Cumulative Intact ^ Effects. 
.5.2.1.0.0. Multiplicative Impact Effects, 
5.2.2.0.0. Transactional Impact Effects. 

5.3.0.0.0. 'Simple Impact/Effectiveness.' 
^S. 3.1. P.O. ^Simple lupact (See Desirability) . 
5.3.2.0.0. Problcm-Sijnple Impact (See Desirability) 



6.0.0/0.0. Efficiency. 




6.1.0.0.0. Cos t/Ef f i cicncy . 
6.1.1.0.0. Continuance. 
6.1.2.0.0. Real Costs 

6. 2 . 0. 0. 0 . Process/Efficiency . 
6:2.1.0.0. Real Time. 
6.2.2.0.0. Time Requirements. 

6.3.0.0.0. Resource/Efficiency. 
6.3.1.0.0. Real Resources. 
6.3.2.0.0, Resource Requirements.' 
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' ^ fositiOR 
WUT POYOU COVSIOEH MIKUKN rumHU FCK tnoCVTlOVAl yKODtC^T 



OmaiONS; In tht ftr Uft colu«ft, ple»»e clrd* your Bijor f<SucattOB product deci^lcn mJc^ rbleCi)/ nen indicate your raiini of Kinima 

criteria. i» Jht •djoinint row(») by »»rkin|: for esi^/ntl«l criteru; o./for drsirsbl/critrrU, ind for unicporuni criteria 
Clrdt key itccs wljhin these crltetii clusters. At botto* of the p»xe ch^i (V) iny clustcn %here you think it U possibU^ 
% tsiA^llsh cordon iiinlnua criteria, / ^ / 'T 
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need 
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values 
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adaptablUty 
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reasonable fiKal 
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availability ( 
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ability to roceive 
any nfeded train 
ixLt to use 
product 



Intrinsic Quality 
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sex, race, ethnic, 
ate* socio-econoaic 
fairness ar balance 
'content accuracy t 
currency 
uniqueness 
initructional 
quality clear pur- 
pose, rationale 
technical quality 
of physical 
features of Mterials 
Ap^al to learners 
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"Spinoffs 



mer verification 
and revision 
quality design 
expert staff 



Cognitive skill, 
perforaance, 
attitJSe, BOtl* 
>vaU0R 

leaminf rate, 
attendanco, 
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others 
credibility 
of evidenct 
support inf 
clalM4 efi*c«r 



sodel for other 
vork • 
contribute to 
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o_ j>H^******y"- 
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o federal '^Jucation * 
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u local education 
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education 
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o organisation f^r 
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EPIE, cititcns, 

FTA. NOW. 
Council on 
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